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Abstract 



Eighteen judges with backgrounds in assessment, decision making, 
and learning disabilities were asked to use an array of infonnation 
to differentiate learning disabled and non-learning disabled students. 
Each judge was provided with forms containing information on 42 test 
or subtest scores of 50 school-identified LD students and 49 non-LD 
students. Judges were extremely inaccurate in their classifications 
and in little agreement with each other. Also, it appeared that dif- 
ferent judges emphasized different factors when making their decisions. 
The results suggest that, given current definitions of the condition 
called "leamxng disabilities." there is little basis for the hope that 
school personnel are going to be able accurately to identify such 
students. 



ERIC 



5 



Inter-Judge Agreement in Classifying 
Students as Learning Disabled 

Traditionally, research in the area of learning disabilities has 
compared learning disabled students to students identified as "educable 
mentally retarded." "emotionally disturbed." or "normal" (Ackerman, 
Peters, & Dykman. 1971; Bussell. Huls. & Long, 1975; Gajar, 1979; 
Wagonseller. 1973; Webster & Schenck, 1978). While numerous attempts 
have been made to identify the salient characteristics that distinguish 
LD students from other students, a major difficulty with such studies is 
that they compare learning disabled students to populations with obviously 
different characteristics. In comparison to the mentally retarded, we 
expect and observe intellectual differences; compared to students classi- 
fied as emotionally disturbed, we expect and find differences in the number 
of behavior problems; in comparison to "normal" students we expect and 
identify differences in the level of academic achievement. We really 
learc very litcle from such investigations. 

Diacnostlc personnel usually have little difficulty distinguishing 
between discernably different kinds of students ("normal." MR. LD) , 
relative to the difficulties they experience in trying to differentiate 
LD students from those who are simply slow learners. Numerous students 
fail to acquire academic skills and fail to meet the objectives that the 
school sets for them. A critical diagnostic problem is seen in efforts 
to decide, from among the population of students who are not "making it" 
in school, who should be declared LD and provided with special educational 
interventions. Research is nearly nonexistent on this critical diagnostic 
problem. 
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Although school personnel typically believe that few low-achieving 
students exist who have not been labeled as LD, Ysseldyke, Algozzine, 
Shinn, and McGue (in press) found a relatively large number of these 
students at the foyrth-grade level. The students scored in the lowest 
quartile on a group achievement test, but had not been identified as LD 
by their school districts. 

Given that the two groups of sf'dents (low achievers and LD) do 
exist, it becomes important to investigate the basis for distinguishing 
them. Ysseldyke et al. (in press) examined the extent to which measures 
of cognitive ability, academic achievement, perceptual-motor skills, self- 
concept, and behavior problems discriminated between low achievers and 
school-identified LD children. Their results indicated considerable 
similarity between the two groups; an average of 96X of the scores were 
within a common range and the performance of the LD and low-achieving 
students on many of the subtests was identical. Based on psychometric 
measures only, Ysseldyke et al. (in press) raised serious concerrs regard- 
ing the differential classification of low-achieving students as either 
learning disabled or non-learning disabled. 

Some school personnel have argued that individual psychometric 
measures do not provide the necessary information for distinguishing 
between LD and low-achieving students. Rather, they maintain that deci- 
sions actually are made on the basis of the pattern of scores that 
students obtain on different psychometric instruments. It thus becomes 
important to ascertain the extent to which school decision makers can 
distinguish between the two groups when given patterns of scores on 
multiple measures. 
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I The use of a number of scores to identify learning disabled 

students is, in fact, mandated by the U.S. Office of Education. The 
August 23, 1977 Federal Register specifically states th^t "no single 
p procedure is used as the sole criterion for determining an appropriate 

educational program for a child" (p. 42497). To detecmine the existence 
of a specific learning disability, "a severe discrepancy between achieve- 
^ ment and intellectual ability" (Federal Register. 1976, p. 65083) must 

be demonstrated. Both of these federal guidelines necessitate the use 
of more than one test in an evaluation of a student, a nrocedure which. 
^ in turn, requires an examination of a student's pattern of scores. 

Although an ability-achievement discrepancy is a typical requirement 
foC\lassification as learning disabled, by no means is there consensus 
that it is a useful concept. Salvia and Ysseldyke (1981) caution that 
before a student's difference scores can be interpreted, it must be 
established that the differences are reliable; differences in standard- 
ization samples for the two tests also should be considered. In practice, 
these cautions frequently are ignored. In addition, a nmiber of studies 
(Algozzine, Forgnone, Mercer, & Trifiletti, 1979; Algozzine, Ysseldyke, 
& Shinn, 1980; Ysseldyke & Algozzine, 1979) argue that learning disabili- 
ties is largely a category of under achievement, thus suggesting that 
the degree of students' achievement deficits also should be considered 
when determining eligibility £or LD placement. 

It is likely that the group of students identified as LD by an 
ability-achievement discrepancy will be differen . from the group identi- 
fied by an achievement deficit; in turn, both of these groups may be dif- 
* ferent from the group identified as LD by school decision-nuiking teams. 

The present study was designed to investigate the extent to which 

<Er!c 8 
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professionals could distinguish between the learning disabled and 
non-learning disabled students identified by each of three definitions 
(actual school classification, ability-achievement discrepancy, achieve- 
ment deficit) on the basis of an array of information similar to that 
typically presented in placement team meetings. 

Method 

Subjects 

The sample consisted of 18 judges, 14 of whom were doctoral students 
in school psychology or special education. Of the other four judges, 
two had master's degrees in education, one had a doctorate in clinical 
psychology, and one had a doctorate in special education. Eleven of 
the judges had regular or special education teaching experience and 
seven were certified school psychologists. All 18 judges had previous 
experience in psychoeducational assessnent and/or placement team decision 
making, plus experienc.: in research on learning disabilities. 
Materials 

Forms containing the chronological age of 99 fourth-graders and in- 
formation on their actual test scores in five domains were prepared. 
The domains and the tests used were: (a) aptitude (WISC-R, Woodcock- 
Johnson Tests of Cognitive Ability), (b) academic achievement (Woodcock- 
Johnson Tests of Achievement, PIAT, Stanford Achievement Test - math 
calculation and math concepts), (c) perceptual-motor (Bender, VMI), (d) 
self-concept (Piers-Harris), and (e) behavior problems (Peterson-Quay). 
Across these five domains, test scores were provided on 42 variables. 

Fifty of the 99 students were school-identified learning disabled 
students and 49 were low achievers from metropolitan Minneapolis and 
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St. Paul schools; this classification was not indicated on the judges' 
test score forms. The exact criteria for identification of students 
as LD by the schools were unknown. The low-achieving group had not been 
identified as LD by their school districts, but scored at or below the 
25th percentile on the Iowa Tests of Basic Skills which had been adminis- 
tered by the schools during the fall of the school year. 

In addition to the schools' classification as a criterion for con- 
sidering a student LD, the federal definition of learning disabilities, 
taken from the December 29, 1977 Federal Register , was used to classify 
students as LD or non-LD. That definition lists seven aieas in which a 
child may be determined to have a specific learning disability: oral 
expression, basic reading skill, reading comprehension, mathematics 
calculation, mathejnatics reasoning, listening comprehension, and written 
expression. However, the federal definition does not specify the amount 
of discrepancy between ability and achievement that is required. In 
the present study, the federal definition was operationalized in two 
ways using subjects' observed aptitude-achievement test scores in the 
areas of intelligence and achievement. The following measures were used 
to ascertain discrepancy in five of the seven areas: (a) written ex- 
pression (WISC-R Full - W-J Written Language Achievement), (b) basic 
reading skill (WI:5C-R Full - W-J Reading Achievement), (c) reading com- 
prehension (WISC-R - PIAT Reading Comprehension), (d) mathematics cal- 
culation (WISC-R Full - Stanford Mathematics Calculation), and (e) 
mathematics reasoning (WISC-R Full - Stanford Marhemat.ics Concepts). In 
one condition (1.0 SD) , the student was considered LD by definition if 
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there was between a 1.0 and 1.5 standard deviation difference on at 
least one of the five aptitude-achievement discrepancy scores (differ- 
ence scores of 15 to 22). In the other condition (1.5 SD) . the student 
was considered LD if there was at least a 1.5 standard deviation 
difference on at least one of the five aptitude-achievement discrepancy 
scores (difference scores of 23 or more). Table 1 indicates the number 
of school-identified LD students and low achievers who were classified as 
LD and non-LD based on the school's definition and the federal definitions, 

Insert Table 1 about here 



Along with the schools' classification and the two operationaliza- 
tions of the federal definition as criteria for considering a student LD. 
the degree of students' achievement deficits was used to classify students 
as LD or non-LD. The achievement deficits were operationalizad in two 
ways using subjects' observed test scores in written expression, basic 
reading skill, reading comprehension, mathematics calculation, and mathe- 
matics reasoning. In one condition (1.0 SD). the student was considered 
LD by definition if the score earned on at least one of the criterion 
measures was between 1.0 and 1.5 standard deviations below the mean (stan- 
dard scores of 78 to 85). In the other condition (-1.5 SD) . the student was 
considered LD if the score earned was at least 1.5 standard deviations 
below the mean (standard scores of 77 or below). Table 2 indicates th-^ 
number of school-identified LD students and low ? .nievers who were 
classified as LD and non-LD bused on the schools' definition and che 
low achievement definitions. 
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Insert Table 2 about here 

Along with the "test scores" form, the packet of materials given to 
judges included a "tests administered" form that specified which tests 
were used in each of the five areas. A third sheet provided normative 
data for those measures that did not have a mean of 100 and standard 
deviation of 15. These i-cluded the BeUer, Visual-Motor Gestalt Test, 
the Developmental Test of Visual-Motor Integration, and the Peterson- 
Quay Behavior Problem Checklist. Also provided were the full group 
mean and standard deviation for the Piers-Harris Children's Self-Concept 
Scale, based upon the total sample of 99 students. 
Procedure 

Each judge was given a packet of materials and a response form, 
and then instructed to examine the test scores for the 99 students and 
to indicate whx>.h he/she believed were learning disabled and which were 
noa-learning disabled. 

Results 

Judges' Accuracy 

Judges' accuracy in idantifying learning disabled students is reported 
in Tables J and 4 . A comparison was made between judges' ratings and school 
classification, between judges' ratings and the classification from each 
of the two federal definitions, and between judges' ratings and the classi- 
fications from each of the two levels of achievement deficit. The accuracy 
percentages were computed separately for LD and non-LD students uithin each 
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of * the five conditions and dependent-sample ^s were calculated. Across 
all five conditions, Judges were more accurate in identifying the non-LD 
than the LD students [school definition: t^ (17) - 3.89, £ < .001; 1.0 
SD federal definition: t^ - 5.23, £ < .001; 1.5 SD federal definition: 
Jt (17) - 3.44, £ < .001; 1.0 aD low achievement definition: t^ (17) - 4.08, 
£ < .001; 1.5 SD low achievement definition: £ (17) - 2.71, p < .01]. 
For the identification of LD students across all five criterion conditions, 
the percentages of judges' accuracy rar^z^- Tirom 2.9 to 72.0; for non-LD 
students, :he percentages of accuracy ranged from 35.4 to 100.0. 

Insert Tables 3 and 4 about here 

Judges' Leniency 

Also examined was the judges' leniency, a factor which reflects the 
percentage of children classified as LD. By definition, a lenient Judge 
classifies more students as LD than asnon-LD. A non<^lenient Judge 
classifies fewer students as LD than asnon-LD. As can be seen in Table 
5, only 4 of the 18 Judges were lenient; that is, they identified 
50Z or more of the students as LD. 



Insert Table 5 about here 

Considering that lenient Judges classify a greater number of students 
as LD, they will have a relatively higher accuracy in identifying the LD 
than in identifying the non-LD students. That is, the four lenient Judges 
(4, 10, 14, and 18) are four of the most accurate Judges in identifying 
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iu studenes (although none of the Judges were in agr ement with the 
schools' classification for as many as three-fourths of the students) 
and are relatively inaccurate in identifying the non-LD students. 
Similarly, the non-lenient Judges, those who classify more, students as 
non-LD will be relatively more accurate in identifying non-LD students, 
but at the expense of their accuracy in identifying LD students. The 
negative relationship between accuracy in identifying LD and non-LD 
students is a dirtct result of the confounding between leniency and 
accuracy. This confounding is best illustrated by considering the 
totally lenient Judge, that is, one who classifies all students as LD. 
Such a Judge would correctly identify all LD students as LD, but would 
Incorrectly identify all non-LD students as LD. Consequently, caution 
is necessary in interpreting . esults of Tables 3 and 4; one must 
take into account the inherent confounding between leniency and accuracy 
of classification. 
Inter- , ^udge Agreement 

A third factor investigated was inter-Judge agreement, the extent 
to which Judges agreed with each other in their ratings of the students. 
In deriving the index of interrater agreement, the point biserial cor- 
relation between each Judge 'c ratings and the average rating of the 
remaining 17 Judges vas first computed. The resultinf' ooint biserial 
correUaon was then divided by the maximum possible point biserial which 
could result given the proportion of children classified as learning 
disabled and non-LD for that given judge. The agreement index 
was defined as the ratio of actual point biserial to the maximum point 
biserial, and can be interpreted as the agreement of that Judge's 
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racings with all other judges' ratings relative to the maximum level 
of agreement possible given that judge's leniency. 

As can be seen In Table 5, there Is a wide range In Inter- judge 
agieement. from .067 (judge 11) to .803 (judge 14). Yet In examining 
their accuracy In Identifying LD and non-LD students, judge 11 and 
judge 14 were very similar In their percentages of agreement with the 
school definition and the 1.5 SD federal definition. In addition, the 
two most accurate Judges In overall agreement with the schools (judges 
14 and 17) were actually considerably different In the extent to which 
each of them agreed with the other 17 judges. Judge 14 was In high 
agreement with the other judges (.803). yet judge 17 was In low agree- 
ment C.319). 

Frequency of Judges' LD Ratings 

For each of the 99 students, a frequency count was made of the 
number of judges who rated the student as LD. For 94 of the students, 
there was at least one judge who rated the student as LD. There were 
no students whom everyone Identified as LD, but there were five students 
whom everyone rated as non-LD. Three of th """ f^v«^ students, however. 
were actually Identified as LD bv the schools . 
Facto. s Influencing Decisions 

The standard procedure adopted by several Investigators to model the 
judges' decision-making process Is the method of di^jcriminant function 
(Shavelson, 1979). This method is preferred since it results in a par- 
simonious and statistically optimal description of the judge's decisions; 
we sought an alternative to it because of two shortcomings in the method. 



ERIC 



11 



First, the discriminant function is simply a statistical model of the deci- 
sion-making process and does not necessarily identify those variables that 
were important in making the decision (Goldberg, 1970). Second, the low 
ratio of subjects (99) to variables (48) here would make the results of any 
discriminant function analysis hiphly unscable. 

As an alternative to the discriminant function, we analyzed the mean 
differences between the judge-identified LD and non-LD students for each of 
the 18 judges. This was accomplished as follows. First, for each judge, 
the mean difference between the j.idge-id^if led LD and non-LD students was 
calculated on each of the A2 original test. scores plus an additional six 
discrepancy scores (WISC-R Verbal - WISC-R Performance, W-J Reading Aptitude - 
W-J Reading- Achievement, W-J Mathematics Aptitude - U-J Mathematics Achievement, 
W-J Written Language Aptitude - W-J Written Language Achievement, WISC-R Full - 
W-J Broad Cognitive, and WISC-R Full - ?IAT Total). The 48 absolute mean 
differences were then rank ordered from largest to smallest. For example, 
for a given judge, the variable ranked ^Ust would be that variable for 
which the students judged learning disabled were most different from the 
students judged non-learning disabled, and thus a variable which apparently 
played a major role in the differentiation of LD from non-LD for that judge. 
Conversely, the variable ranked last would be that variable for which the 
students judged LD were most similar to the students judged non-LD, and thus 
of no apparent importance in differentiating the two groups. 

Considering that all four lenient juages were more accurate in identifying 
LD students and that the 14 non-leaienr judges were more accurate in identifying 
non-LD students, a Judge's leniency is an apparent factor in the accuracy of 
the ratings. In order to determine the extent to which the differences between 
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the lenient and the non-lenlent judges could be a function of their using dlf- 
ferent variables to arrive at a decision, we calculated the average rank for 
each of the AS measures for the lenient and non-lenlent judges. Table 6 
lists the 10 .ost important varlahles. as judged by average rank, for the len- 
lent and non-lenlent judpes. 



Insert Table 6 about here 
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AS can be seen In Table 6. the lenient judges were In very high agreement 
regarding the Importance of an aptitude-achievement discrepancy (as measured 
by the WISC-R Full scale IQ minus the FIAT Total Score); the average ranking 
„a, 2.00. The rankings for the non-lenlent judges also Indicated the ^.mportance 
of the aptitude-achievement discrepancy, although they were not In as high agree- 
ment; the variable had an average ranking of 13.79. Of the 10 ranked variables 
listed for each group of iudges. 5 of them are common to both the lenient and 
aon-lenlent judges, namely, the aptitude-achievement discrepancy and A 
achievement measures. Overall, the rank ordering of variables suggests that 
both lenient and non-lenlent judges as a group gave more consideration to 
measures of discrepancy and achievement than to measures of scholastic ap- 
titude, perceptual-motor skills, or behavior problems. 

At the gi;oup level, judges apparently were emphasizing aptitude-achieve- 
ment discrepancy and achievement when differentiating LD from non-LD students. 
Yet the salient factors that Individual judges used in making their decisions 
remain undellneated. To examine the variables that individual judges used 
to classify the students, mean difference scores were examined. For each 
judge, the means of each of the A2 test or subtest s- res were calculated for 
students whom Che judge classified as LD and for students whom the judge 
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classified as non-LD. Thus, the means indicated the average scores for 
particular variables for the judge-identified LD and non-LD groups. At the 
idiographic level of analysis, eight case studies are presented and include the 
two most lenient and two least lenient judges and the two nost accurate and 
two least accurate judges (compared with the schools' ratings). The variables 
t\^at they used to classify students, in order of magnitude, are presented in 
Table 7 



Insert Table 7 about here 

Most Lenient Judges 

Case n . The most lenient judge (judge 10) classified 60.6Z of the 
students as LD. The mean difference scores suggested that this judge primar- 
ily used aptitude-achievement and verbal-performance discrepancies to differ- 
entiate the LD from the non-LD students. 

Case n . The second most lenient judge (judge 14) classified 52.5% of 
the students as LD. This judge also seemed to examine aptitude-achievement 
discrepancy, but placed more emphasis on achievement measures. 

Least Lenient Judges 

Case B. The least lenient judge (judge 9) only classified 5.1Z of the 
students as LD. Mean difference scores suggest that school-related aptitude 
scores were the primary consideration. Behavior ratings and an achievement 
measure apparently were examined as well. 

Case U. The second least lenient judge (judge 2) classified 11.1% of 
the students as LD. In a similar manner as the most lenient judges, this iudpe 
also emphasized aptitude-achievement discrepancy together with cognitive 
measures. 
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Moat Accurate Judges 

Case #5 . One of the most accurate judges (judge 17) was In overall agree- 
oent with the schools' classification 70.7Z of the time. This judge was 
considerably more accurate in identifying the non-LD (87.851) than in identifying 
the LD (54.0%) students. As for several other judges, aptitude-achievement 
discrepancy apparently was exanined. Cognitive measures and a school-related 
aptitude score also were emphasized with less consideration given to achievement 
measures* 

Case /te . The other most accurate judge (judge 14), who was also one of 
the most lenient judges, was in overall agreement with the schools' classifi- 
cations 70.77, of the time and was generally equally accurate in identifying 
both the LD and the non-LD students. This judge apparently emphasized aptitude- 
^.chievement discrepancy, as did the other highly accurate judge, but gave more 
consideration to achievement measures. 
Least Accurate Judges 

Case One of the least accurate judges (judge 1) was in overall 

agreement with the schools' classifications 52. 6Z of the time. Aptituxie- 
achievement discrepancy and discrepancy measures apparently were emphasized. 

Case ifS . The other least accurate judge (judge 3) was in overall 
agreement with the schools 52.5% of the time. Aptitude-achievement dis- 
<:repancy was examined. Unlike the other least accurate judge, however, cog- 
nitive measures, rather than achievement measures, seemed to have been given 
strong consideration. 

Discussion 

Clasaification of students as learning disabled is clearly problematic. 
Ysseidyke et al. (in press) provided evidence that there are no meaningful 
psychoip' trie differences between students identified by schools as learning 
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disabled and those who are low achievers and not Identified as LD. They 
reported a 96Z overlap between the two groups In performance on psychometric 
measures. Psychologists and educators who have viewed these results have 
argued repeatedly that while there Is considerable overlap between the two 
groups, they can, given pupils' pattern of scores on numerous devices, dif- 
ferentiate between low achievers and those who are learning disabled. This 
argument, often called the "I know one when I see one" argument, was tested 
In this study. 

Judges were provided with the scores earned on 42 tests or subtests 
by 50 school- Identified LD students and 49 non-LD students. Whan asked to 
discriminate between the LD and non-LD students, judges were extremely In- 
accurate (whether compared to the school's classlf i-.atlon, the classifications 
based upon the two federal definitions, or the two low achievement definitions) 
and in little agreement with each other. They also emphasized different, 
factors. Almost all of the students were labeled LD by at least one judge, 
and only five were consistently labeled by all judges as non-LD (despite 
the fact that three of these actually were identified as LD by the schools). 
Given current definitions of the condition called "learning disabilities," 
there is little basis for the hope that school personnel are going to be 
able accurately to identify such students. Of course, the judges in the 
present study were not currently within the educational systeu and were not 
currently making decisions abouc the classification of students. A final 
test would be to have actual decision makers attempt to use patterns in the 
data to differentiate LD and low-achieving students- 
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Table 1 

Number of Students within Categories of School and 
Federal Definitions of Learning Disabilities 



Federal Definition^ School Definition 

LD Non-LD 

— 9 '. 

Non-LD (1-14) 14 

1.0 SD LD (15-22) 18 20 

1.5 SD LD (23 or more) 19 15 

^Numbers in parentheses indicate the ability-achievement discrepancy 
s^rts required for each operationalization of the federal definition. 



3 



Table 2 

Number of Studtnts Within Categories of School and 
Low Achievement Definitions of Learning Disabilitie 



Low Achievement Definition^ School Definition 

LD Non-LD 



Non-LD (86 or above) H 

1.0 SD LD (78 to 85) 22 23 

1.5 SD LD (77 or below) 17 7 



Vnnbers in parentheses indicate the achievement deficit scores 
required for each operationalization of the low achievement 
definition. 



Judge's Accuracy in Identifying LD and 



School Definition 



Judge No. 


X Correct 
LD 


X Correct 
Non-LD 


X Overall 
Accuracy 


1 


• U 


71 A 


52 .6 


*2 




7O . u 


58.6 


3 


OA A 


ox . D 


52 .5 




liO A 




60*6 


5 


1 A "7 


Ql 7 


62.9 


6 


O O A 

ZZ • u 


Ql ft 


S6 6 


7 


44 .0 


Q 1 

oi . 0 


ft7 ft 


8 


CO A 

3Z . 0 


0 J . J 


/ . u 


*9 


1 A A 

10 .0 


1 AA A 
iUU •U 


S4 6 


1^10 


A 

bo • U 


A A Q 


57 .5 


i 1 


64 .0 


65. 3 


64.6 


12 


52.0 


59.2 


55.6 


13 


22.0 


87.8 


54.5 


#14 


"^2.0 


69.4 


70.7 


15 


50.0 


59.2 


54.6 


16 


46.0 


67.3 


56.5 


17 


54.0 


87.8 


70.7 


118 


65.3 


61.7 


63.5 




t (17) 


- 3.89. ^ < 


: .001 



* ^ least lenient Judges 

# « lenient judges 



Table 3 

>n-LD Students According to School and Federal Definitions 



O 



1.0 SD Federal Definition 

X Correct X Correct X OvercU 
LD Non-LD Accuracy 



36.1 


81.5 


48.5 


15.3 


100.0 


38.4 


27.8 


96.3 


46.5 


63.9 


81.5 


68.7 


18.3 


69.2 


32.0 


18.1 


92.6 


38.4 


33.3 


74.1 


44.4 


51.4 


74.1 


57.6 


4.2 


92.6 


28,3 


70.8 


66.7 


69.7 


37.5 


18.5 


32.4 


56.9 


81.5 


63.6 


22.2 


96.3 


42.5 


63.9 


77.8 


67.7 


55.6 


81 .5 


62.6 


40.3 


63.0 


46.5 


40.3 


85.2 


52.5 


55.1 


55.6 


55.2 



1.5 SD Federal Definition 

% Correct X Correct X Overall 
LD Non-LD Accuracy 



44.1 


75.4 


64.7 


26.5 


96.9 


72.7 


35.3 


86.2 


68.7 


70.6 


58.5 


62.6 


14.7 


74.6 


53.7 


20.6 


87.7 


64.7 


35.3 


70.8 


58.6 


61.8 


64.6 


63.6 


2.9 


93.8 


62.6 


82.4 


50.8 


61 .6 


20.6 


35.4 


30.3 


61.8 


61.5 


61.6 


26.5 


87.7 


66.7 


64.7 


53.8 


57.6 


61.8 


63.1 


62.6 


41.2 


61.5 


54.5 


41.2 


70.8 


60.6 


47.1 


45.2 


45. 9 



t (17) - 5.23, £ < .001 



t (17) - 3.44, 2 -001 

Jo 



Judge's Accuracy in Identifying LD and Hon 



Table 4 

-LD Students According to School and Low Achievement Definitions 



Judge No. 
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School Definition 



% Correct % Correct X Overall 
LD Mon-LH Accuracy 



1 


'XL n 


71.4 


52.6 


*2 


) r\ n 
lu • U 




58.6 


3 


OA A 


U X • u 


52.5 


«4 


OZ • U 




60.6 


5 


34 . 7 


Q 1 7 


62.9 


6 


22.0 




- 6.6 


7 


44.0 


81.6 


62.6 


8 


52.0 


63.3 


57.6 


*9 


10.0 


100.0 


54.6 


tno 


68.0 


46.9 


57.5 


11 


64.0 


65.3 


64.6 


12 


52.0 


59.2 


55.6 


13 


22.0 


87.8 


•:4.5 


tflA 


72.0 


69.4 


70.7 


15 


50. 0'*" 


59.2 


54.6 


16 


46.0 


67.3 


56.5 


17 


54.0 


87.8 


70.7 


/.'18 


65.3 


61.7 


63.5 




t (17) - 3. 


89. P < -001 





least lenient Judges 
ienien. fudges 



1.0 SD Low Achievement Definition 

% Correct 2 Correct X Overall 
LD Non-LH Accuracy 



1.5 SD Low Achievement Definition 

% Correct X Correct X Overall 
LD Non-1,n Accuracy 



41.4 


93.1 


56.6 


A 1 1 

^i • / 


7? 0 

/ & • LI 


64.6 


15.7 


100.0 


40.4 


iZ • D 


89 3 


70. 7 


27.1 


93.1 


46.5 


2d .0 


An n 


66.7 


60.0 


69.0 


62.6 


00 • 7 


SI 1 


56.6 


30.4 


100.0 


50.5 


JO • D 


M 7 

07 • ^ 


81.4 


20.0 


96.6 


42.4 


16. 7 


AS 'K 

OD • J 


68.6 


32.9 


72.4 


44.4 


29.2 


u o • u 


58.6 


62 9 


100.0 


73.7 


66.7 


62.7 


63.7 


7.1 


100.0 


34.4 


16.7 


93.7 


78.7 


67.1 


55.2 


63.7 


62.5 


40.0 


45.5 


51.4 


55.2 


52.6 


70.8 


57.3 


5i.6 


42.9 


44.8 


43.4 


41.7 


52.0 


49.5 


24.3 


100.0 


46.5 


20.8 


84.0 


68.4 


62.9 


75.9 


66.6 


70.8 


54.7 


58.4 


52.9 


72.4 


58.6 


54.2 


57.3 


56.5 


50.0 


86.2 


60.7 


50.0 


64.0 


60.6 


42.9 


89.7 


56.6 


45.8 


70.7 


64.6 


=0.0 


42.9 


47.9 


52.2 


47.9 


49.0 


(17) - 


4.08, P < 


.001 


I (17) - 


2.71, P < 


.01 
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Tabic 5 

Individual Judges' Leniency and Agreement with Other Judges 



Inter-Judge 
Leniency Agreement 



1 


31.3 


.644 


2 


11.1 


.713 


3 




22'> 




51.5 


.657 


5 


21.6 


.418 


6 


15.2 


.405 


7 


31.3 


.256 


8 


h4.4 


.700 


9 


5.1 


.275 


#10 


60.6 


.669 


11 


49.5 


.067 


12 


46.5 


.271 


13 


17.2 


.340 


#14 


52.5 


.803 


15 


45.5 


.575 


16 


39.4 


.410 


17 


33.3 


.319 


118 


50.5 


.333 



«Unient Judges 



2:) 



Table 6 

Rank Ordering of the Average Mean Differences Between 
LD and Non-LD Students for Lenient and Non-Lenient Judges* 



23 



Lenient 



Mean 
Rank 



Non-Lenient 



Mean 
Rank 



WISC-R Full - PIAT Total 


2.00 


WISC-R Comprehension 


.13.14 


PIAT Spelling 


7.38 


WISp-R ruii rj_fti lotax 


13.79 


W-J Basic Skills 


7.75 


W-J Basic Skills 


14.07 


PIAT Read Recog 


8.75 


WISC-R Similarities 


15.18 


Picture Arrangemenc 


11.38 


PIAT Raadlng Comp 


16.07 


W-J Math Achiev 


12.38 


W-J Reading Achiev 


16.32 


PIAT Total 


13.50 


W-J Knowledge Apt 


17.68 


W-J Reading Achiev 


U.25 


WISC-R ObJ Assembly 


18.07 


PIAT Reading Comp 


14.50 


PIAT Spelling 


13.18 


WISC-R Verbal-Performance 


14.75 


WISC-R Vocabulary 


18.39 



*The ranking of variables ranged from 1 to 48. The table presents the 
first 10 variables for the two groups of Judges, starting with the 



one of greatest average mean difference. 
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Table 7 

Variables Used by Judges Co Classify Students 



Judge 1 

Picture ArrangetBent (WISC-R subtest) 
WISC-R Full - PIAT Total 
W*J Skills Achievement 
PIAT Reading Comprehension 
PIAT Reading Recognition 

Judge 3 

WISC-R Full - PUT Total 
WISC-R Full 
W-J Reasoning 

Similarities (WTSC-R subtest) 
Picture Arrangement (WISC-R subtest) 

Judge 10 

Picture Arrangement (WISC~R subtest) 

WISC-R Full - PIAT Total 

WISC-R Verbal - WlSC-R Performance 

WISC-R Performance 

W-J Skills Achievement 



Judge 2 

WISC-R Full - PIAT Total 

Object Assembly (WISC-R subtest) 

Similarities (WISC-R subtest) 

IiTISC-R Performance 

Block Design (If[SC-R subtest) 

Judge 9 

W-J Knowledge Aptitude 

W-J Reading Aptitude 

W-J Written Language Aptitude 

Peterson-Quay Total 

PIAT Reading Comprehension 

Judge 14 

W-J Memory 

WISC-R Full - PIAT Total 
W-J Skills Achievement 
PIAT Spelling 



Judge 17 

WISC-R Full - PIAT Total , 
W-J Skills Achievement 
W-J Memory 

WISC-R Full - W-J Broad Cognitive 
W-J Written Language Aptitude 
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